Application Monologue for Self-Driving Vehicles

ABSTRACT

The technology includes communicating the current status of a self-driving vehicle to users, such as passengers within the vehicle and other users awaiting pickup. Certain information about the trip and vehicle status is communicated depending on where the passenger is sitting within the vehicle or where the person awaiting pickup is located outside the vehicle. This includes disseminating the “monologue” of a vehicle operating in an autonomous driving mode to a user via an app on the user&#39;s device (e.g., mobile phone, tablet or laptop PC, wearable, or other computing device) and/or an in-vehicle user interface. The monologue includes current status information regarding driving decisions and related operations or actions. This alerts the user as to why the vehicle is taking (or not taking) a certain action, which reduces confusion and allows the user to focus on other matters.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/710,113, filed on Dec. 19, 2019, the disclosure of which isincorporated herein by reference.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers from onelocation to another. Such vehicles may operate in a fully autonomousmode without a person providing driving input. In this driving mode, thevehicle may make a variety of driving decisions. However, a passenger ora person awaiting pickup may not be aware of why the vehicle isoperating in a certain manner. This can cause confusion and become adistraction.

BRIEF SUMMARY

The technology relates to communicating the current status of aself-driving vehicle to users (customers), including passengers withinthe vehicle and users awaiting pickup. Certain information may bepresented to passengers and other users via vehicle display components(e.g., internal display screens or external signage). The type(s) ofinformation and how it is communicated may depend on where the passengeris sitting within the vehicle (e.g., front seat v. rear seat) or wherethe person awaiting pickup is located outside the vehicle.

According to one aspect, a method of operation comprises determining, byone or more processors of a vehicle operating in an autonomous drivingmode, status information associated with the vehicle; determining, bythe one or more processors, a pickup status of a customer, the pickupstatus indicating whether the customer is awaiting pickup by the vehicleor is currently being driven by the vehicle in the autonomous drivingmode; selecting, by the one or more processors, one of a plurality ofcommunication options for communication with the customer based on thepickup status; generating, by the one or more processors in accordancewith the selected communication option, a monologue message indicatingstatus information associated with the vehicle; and providing themonologue message for presentation to the customer.

The status information may be a current driving status of the vehicle,in which the monologue message indicates the current driving status. Forinstance, the current driving status may be that the vehicle is waitingto perform a driving action based on an event or activity in an externaldriving environment. Here, the vehicle may be waiting to perform thedriving action either in response to a traffic signal status or whileobeying a traffic signal.

In one example, when the pickup status indicates that the customer isawaiting pickup, providing the monologue message for presentation to thecustomer includes transmitting the monologue message from the vehicle toa user device of the customer. In another example, when the pickupstatus indicates that the customer is being driven by the vehicle,providing the monologue message for presentation to the customerincludes generating at least one of visual or acoustical information viaan in-vehicle user interface system. Providing the monologue message forpresentation to the customer may further include transmitting themonologue message from the vehicle to a user device of the customer.Alternatively or additionally, the method may also include receiving aquery from the customer based on presentation of the monologue messageand, in response to the query, providing additional information to thecustomer via the in-vehicle user interface system. Here, the additionalinformation may include contextual text, imagery or audible informationregarding the query for the monologue message.

In a further example, generating the monologue message includesselecting graphical information for presentation to the customer basedon a ranked list of monologue data. The list of monologue data may beranked based on a hierarchical order of (i) features that make thevehicle stop driving, (ii) features that the vehicle predicts will causea pause in driving exceeding a threshold amount of time, (iii) featuresthat cause the vehicle to move at lower than a posted speed, and (iv)features that may cause the vehicle to deviate from a planned course ofaction. Alternatively or additionally, selecting based on the rankedlist may be based on at least one of time or distance informationassociated with a current or upcoming trip by the customer in thevehicle.

And in yet another example, when the pickup status indicates that thecustomer is being driven by the vehicle, providing the monologue messagefor presentation to the customer includes selecting monologue messagedetails based on where in the vehicle the customer is sitting.

According to another aspect, a method of operation includes determining,by one or more processors, status information associated with a vehicleoperating in an autonomous driving mode; determining, by the one or moreprocessors, a pickup status of a customer, the pickup status indicatingwhether the customer is awaiting pickup by the vehicle or is currentlybeing driven by the vehicle in the autonomous driving mode; selecting,by the one or more processors based on the pickup status, one of aplurality of communication options for communication with the customervia a personal communication device of the customer; generating, by theone or more processors in accordance with the selected communicationoption, a monologue message indicating status information associatedwith the vehicle; and providing the monologue message for presentationto the customer via the personal communication device.

In one example, a first one of the one or more processors is aprocessing device of the vehicle. In another example, a first one of theone or more processors is a processing device of a remote server inoperative communication with the vehicle. Here, a second one of the oneor more processors may be a processing device of the vehicle. In thiscase, the method further comprises: the first processor generating afirst set of information for the monologue message; and the secondprocessor generating a second set of information for the monologuemessage.

In a further example, a first one of the one or more processors is aprocessing device of the personal communication device. In this case,the method further comprises the processing device selecting informationreceived from at least one of the vehicle or a remote server forpresentation in the monologue message.

In one scenario, when the pickup status indicates that the customer isbeing driven by the vehicle, selecting one of the plurality ofcommunication options further includes selecting to present themonologue message to the customer via an in-vehicle user interfacesystem in addition to presenting via the personal communication device.This process may further comprise providing additional information tothe customer via the in-vehicle user interface system, in which theadditional information includes contextual text, imagery or audibleinformation regarding a query about the monologue message.

And in another example, the status information is a current drivingstatus of the vehicle, and the monologue message indicates the currentdriving status.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-B illustrate an example passenger-type vehicle configured foruse with aspects of the technology.

FIGS. 1C-D illustrate an example articulated bus arrangement for usewith aspects of the technology.

FIG. 2 is a block diagram of systems of an example vehicle in accordancewith aspects of the technology.

FIG. 3 illustrates an exemplary detailed map in accordance with aspectsof the technology.

FIGS. 4A-B illustrate different map examples in accordance with aspectsof the technology.

FIGS. 4C-D illustrate different in-vehicle map displays in accordancewith aspects of the technology.

FIGS. 5A-B illustrate user device graphical interfaces in accordancewith aspects of the technology.

FIGS. 6A-D illustrate examples of different message presentations inaccordance with aspects of the technology.

FIGS. 7A-B illustrates an example system in accordance with aspects ofthe technology.

FIG. 8 illustrates an example method in accordance with aspects of thetechnology.

FIG. 9 illustrates another example method in accordance with aspects ofthe technology.

DETAILED DESCRIPTION

Aspects of the technology involve disseminating the “monologue” of avehicle operating in an autonomous driving mode to a user via an app onthe user's device (e.g., mobile phone, tablet or laptop PC, wearable, orother computing device) and/or an in-vehicle user interface. Asdiscussed further below, the monologue includes current statusinformation regarding driving decisions and related operations oractions. This alerts the user as to why the vehicle is taking (or nottaking) a certain action, which reduces confusion and allows the user tofocus on other matters.

Example Vehicle Systems

FIG. 1A illustrates a perspective view of an example passenger vehicle100, such as a minivan, sport utility vehicle (SUV) or other vehicle.FIG. 1B illustrates a top-down view of the passenger vehicle 100. Asshown, the passenger vehicle 100 includes various sensors for obtaininginformation about the vehicle's external environment, which enable thevehicle to operate in an autonomous driving mode. For instance, aroof-top housing 102 may include a lidar sensor as well as variouscameras, radar units, infrared and/or acoustical sensors. Housing 104,located at the front end of vehicle 100, and housings 106 a, 106 b onthe driver's and passenger's sides of the vehicle, may each incorporatelidar, radar, camera and/or other sensors. For example, housing 106 amay be located in front of the driver's side door along a quarter panelof the vehicle. As shown, the passenger vehicle 100 also includeshousings 108 a, 108 b for radar units, lidar and/or cameras also locatedtowards the rear roof portion of the vehicle. Additional lidar, radarunits and/or cameras (not shown) may be located at other places alongthe vehicle 100. For instance, arrow 110 indicates that a sensor unit(112 in FIG. 1B) may be positioned along the rear of the vehicle 100,such as on or adjacent to the bumper. And arrow 114 indicates a seriesof sensor units 116 arranged along a forward-facing direction of thevehicle. In some examples, the passenger vehicle 100 also may includevarious sensors for obtaining information about the vehicle's interiorspaces (not shown).

FIGS. 1C-D illustrate an example of another type of vehicle 120, such asan articulated bus. As with the passenger vehicle 100, the articulatedbus 120 may include one or more sensor units disposed along differentareas of the vehicle.

By way of example, each sensor unit may include one or more sensors,such as lidar, radar, camera (e.g., optical or infrared), acoustical(e.g., microphone or sonar-type sensor), inertial (e.g., accelerometer,gyroscope, etc.) or other sensors (e.g., positioning sensors such as GPSsensors). While certain aspects of the disclosure may be particularlyuseful in connection with specific types of vehicles, the vehicle may beany type of vehicle including, but not limited to, cars, trucks,motorcycles, buses, recreational vehicles, etc.

There are different degrees of autonomy that may occur for a vehicleoperating in a partially or fully autonomous driving mode. The U.S.National Highway Traffic Safety Administration and the Society ofAutomotive Engineers have identified different levels to indicate howmuch, or how little, the vehicle controls the driving. For instance,Level 0 has no automation and the driver makes all driving-relateddecisions. The lowest semi-autonomous mode, Level 1, includes some driveassistance such as cruise control. Level 2 has partial automation ofcertain driving operations, while Level 3 involves conditionalautomation that can enable a person in the driver's seat to take controlas warranted. In contrast, Level 4 is a high automation level where thevehicle is able to drive fully autonomously without human assistance inselect conditions. And Level 5 is a fully autonomous mode in which thevehicle is able to drive without assistance in all situations. Thearchitectures, components, systems and methods described herein canfunction in any of the semi or fully-autonomous modes, e.g., Levels 1-5,which are referred to herein as autonomous driving modes. Thus,reference to an autonomous driving mode includes both partial and fullautonomy.

FIG. 2 illustrates a block diagram 200 with various components andsystems of an exemplary vehicle, such as passenger vehicle 100 or bus120, to operate in an autonomous driving mode. As shown, the blockdiagram 200 includes one or more computing devices 202, such ascomputing devices containing one or more processors 204, memory 206 andother components typically present in general purpose computing devices.The memory 206 stores information accessible by the one or moreprocessors 204, including instructions 208 and data 210 that may beexecuted or otherwise used by the processor(s) 204. The computing systemmay control overall operation of the vehicle when operating in anautonomous driving mode.

The memory 206 stores information accessible by the processors 204,including instructions 208 and data 210 that may be executed orotherwise used by the processors 204. The memory 206 may be of any typecapable of storing information accessible by the processor, including acomputing device-readable medium. The memory is a non-transitory mediumsuch as a hard-drive, memory card, optical disk, solid-state, etc.Systems may include different combinations of the foregoing, wherebydifferent portions of the instructions and data are stored on differenttypes of media.

The instructions 208 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor(s). For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions”, “modules” and “programs” may be usedinterchangeably herein. The instructions may be stored in object codeformat for direct processing by the processor, or in any other computingdevice language including scripts or collections of independent sourcecode modules that are interpreted on demand or compiled in advance. Thedata 210 may be retrieved, stored or modified by one or more processors204 in accordance with the instructions 208. In one example, some or allof the memory 206 may be an event data recorder or other secure datastorage system configured to store vehicle diagnostics and/or obtainedsensor data, which may be on board the vehicle or remote, depending onthe implementation.

The processors 204 may be any conventional processors, such ascommercially available CPUs. Alternatively, each processor may be adedicated device such as an ASIC or other hardware-based processor.Although FIG. 2 functionally illustrates the processors, memory, andother elements of computing devices 202 as being within the same block,such devices may actually include multiple processors, computingdevices, or memories that may or may not be stored within the samephysical housing. Similarly, the memory 206 may be a hard drive or otherstorage media located in a housing different from that of theprocessor(s) 204. Accordingly, references to a processor or computingdevice will be understood to include references to a collection ofprocessors or computing devices or memories that may or may not operatein parallel.

In one example, the computing devices 202 may form an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system is configured to communicate with various components ofthe vehicle. For example, the computing devices 202 may be incommunication with various systems of the vehicle, including a drivingsystem including a deceleration system 212 (for controlling braking ofthe vehicle), acceleration system 214 (for controlling acceleration ofthe vehicle), steering system 216 (for controlling the orientation ofthe wheels and direction of the vehicle), signaling system 218 (forcontrolling turn signals), navigation system 220 (for navigating thevehicle to a location or around objects) and a positioning system 222(for determining the position of the vehicle, e.g., including thevehicle's pose). The autonomous driving computing system may employ aplanner module 223, in accordance with the navigation system 220, thepositioning system 222 and/or other components of the system, e.g., fordetermining a route from a starting point to a destination or for makingmodifications to various driving aspects in view of current or expectedtraction conditions.

The computing devices 202 are also operatively coupled to a perceptionsystem 224 (for detecting objects in the vehicle's environment), a powersystem 226 (for example, a battery and/or gas or diesel powered engine)and a transmission system 230 in order to control the movement, speed,etc., of the vehicle in accordance with the instructions 208 of memory206 in an autonomous driving mode which does not require or needcontinuous or periodic input from a passenger of the vehicle. Some orall of the wheels/tires 228 are coupled to the transmission system 230,and the computing devices 202 may be able to receive information abouttire pressure, balance and other factors that may impact driving in anautonomous mode.

The computing devices 202 may control the direction and speed of thevehicle, e.g., via the planner module 223, by controlling variouscomponents. By way of example, computing devices 202 may navigate thevehicle to a destination location completely autonomously using datafrom the map information and navigation system 220. Computing devices202 may use the positioning system 222 to determine the vehicle'slocation and the perception system 224 to detect and respond to objectswhen needed to reach the location safely. In order to do so, computingdevices 202 may cause the vehicle to accelerate (e.g., by increasingfuel or other energy provided to the engine by acceleration system 214),decelerate (e.g., by decreasing the fuel supplied to the engine,changing gears, and/or by applying brakes by deceleration system 212),change direction (e.g., by turning the front or other wheels of vehicle100 by steering system 216), and signal such changes (e.g., by lightingturn signals of signaling system 218). Thus, the acceleration system 214and deceleration system 212 may be a part of a drivetrain or other typeof transmission system 230 that includes various components between anengine of the vehicle and the wheels of the vehicle. Again, bycontrolling these systems, computing devices 202 may also control thetransmission system 230 of the vehicle in order to maneuver the vehicleautonomously.

Navigation system 220 may be used by computing devices 202 in order todetermine and follow a route to a location. In this regard, thenavigation system 220 and/or memory 206 may store map information, e.g.,highly detailed maps that computing devices 202 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways, lane markers, intersections, crosswalks,speed limits, traffic signal lights, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. The lanemarkers may include features such as solid or broken double or singlelane lines, solid or broken lane lines, reflectors, etc. A given lanemay be associated with left and/or right lane lines or other lanemarkers that define the boundary of the lane. Thus, most lanes may bebounded by a left edge of one lane line and a right edge of another laneline.

The perception system 224 includes sensors 232 for detecting objectsexternal to the vehicle. The sensors 232 are located in one or moresensor units around the vehicle. The detected objects may be othervehicles, obstacles in the roadway, traffic signals, signs, trees,bicyclists, pedestrians, etc. The sensors 232 may also detect certainaspects of weather or other environmental conditions, such as snow, rainor water spray, or puddles, ice or other materials on the roadway.

By way of example only, the perception system 224 may include one ormore light detection and ranging (lidar) sensors, radar units, cameras(e.g., optical imaging devices, with or without a neutral-density filter(ND) filter), positioning sensors (e.g., gyroscopes, accelerometersand/or other inertial components), infrared sensors, acoustical sensors(e.g., microphones or sonar transducers), and/or any other detectiondevices that record data which may be processed by computing devices202. Such sensors of the perception system 224 may detect objectsoutside of the vehicle and their characteristics such as location,orientation, size, shape, type (for instance, vehicle, pedestrian,bicyclist, etc.), heading, speed of movement relative to the vehicle,etc.

The perception system 224 may also include other sensors within thevehicle to detect objects and conditions within the vehicle, such as inthe passenger compartment. For instance, such sensors may detect, e.g.,one or more persons, pets, packages, etc., as well as conditions withinand/or outside the vehicle such as temperature, humidity, etc. This caninclude detecting where the passenger(s) is sitting within the vehicle(e.g., front passenger seat versus second or third row seat, left sideof the vehicle versus the right side, etc.). The interior sensors maydetect the proximity, position and/or line of sight of the passengers inrelation to one or more display devices of the passenger compartment.Still further sensors 232 of the perception system 224 may measure therate of rotation of the wheels 228, an amount or a type of braking bythe deceleration system 212, and other factors associated with theequipment of the vehicle itself.

The raw data obtained by the sensors can be processed by the perceptionsystem 224 and/or sent for further processing to the computing devices202 periodically or continuously as the data is generated by theperception system 224. Computing devices 202 may use the positioningsystem 222 to determine the vehicle's location and perception system 224to detect and respond to objects when needed to reach the locationsafely, e.g., via adjustments made by planner module 223, includingadjustments in operation to deal with occlusions and other issues. Inaddition, the computing devices 202 may perform calibration ofindividual sensors, all sensors in a particular sensor assembly, orbetween sensors in different sensor assemblies or other physicalhousings.

As illustrated in FIGS. 1A-B, certain sensors of the perception system224 may be incorporated into one or more sensor assemblies or housings.In one example, these may be integrated into the side-view mirrors onthe vehicle. In another example, other sensors may be part of theroof-top housing 102, or other sensor housings or units 106 a,b, 108a,b, 112 and/or 116. The computing devices 202 may communicate with thesensor assemblies located on or otherwise distributed along the vehicle.Each assembly may have one or more types of sensors such as thosedescribed above.

Returning to FIG. 2 , computing devices 202 may include all of thecomponents normally used in connection with a computing device such asthe processor and memory described above as well as a user interfacesubsystem 234. The user interface subsystem 234 may include one or moreuser inputs 236 (e.g., a mouse, keyboard, touch screen and/ormicrophone) and one or more display devices 238 (e.g., a monitor havinga screen or any other electrical device that is operable to displayinformation). In this regard, an internal electronic display may belocated within a cabin of the vehicle (not shown) and may be used bycomputing devices 202 to provide information to passengers within thevehicle. By way of example, displays may be located, e.g., along thedashboard, on the rear of the front row of seats, on a center consolebetween the front row seats, along the doors of the vehicle, extendingfrom an armrest, etc. Other output devices, such as speaker(s) 240 mayalso be located within the passenger vehicle.

The passenger vehicle also includes a communication system 242. Forinstance, the communication system 242 may also include one or morewireless configurations to facilitate communication with other computingdevices, such as passenger computing devices within the vehicle,computing devices external to the vehicle such as in another nearbyvehicle on the roadway, and/or a remote server system. The networkconnections may include short range communication protocols such asBluetooth™, Bluetooth™ low energy (LE), cellular connections, as well asvarious configurations and protocols including the Internet, World WideWeb, intranets, virtual private networks, wide area networks, localnetworks, private networks using communication protocols proprietary toone or more companies, Ethernet, WiFi and HTTP, and various combinationsof the foregoing.

While the components and systems of FIG. 2 are generally described inrelation to a passenger vehicle arrangement, as noted above thetechnology may be employed with other types of vehicles, such as thearticulate bus 120 of FIGS. 1C-D. In this type of larger vehicle, theuser interface elements such as displays, microphones and speakers maybe distributed so that each passenger has his or her own informationpresentation unit and/or one or more common units that can presentstatus information to larger groups of passengers.

Example Implementations

In view of the structures and configurations described above andillustrated in the figures, various aspects will now be described inaccordance with aspects of the technology.

A self-driving vehicle, such as a vehicle with level 4 or level 5autonomy that can perform driving actions without human operation, hasunique requirements and capabilities. This includes making drivingdecisions based on a planned route, received traffic information, andobjects in the external environment detected by the onboard sensors.However, in many instances the in-vehicle passengers and users awaitingpickup may desire status updates or other information from the vehicleabout what the vehicle is currently doing. For instance, a user waitingfor the vehicle to arrive may see a representation of the vehiclepresented on a map. Here, if the vehicle is not moving along the roadwaythe user may wonder what is going on. This type of situation may befrustrating to the user because he or she does not understand what thevehicle is doing. Aspects of the technology can combine information fromthe perception system (e.g., detected objects including signage andlights), stored maps (e.g., roadgraph), and planned actions (e.g., fromthe planner module) to generate timely and relevant messages to the uservia an app on the user's device. Such information may also oralternatively be presented to one or more passengers using displays andspeakers that are disposed in the passenger compartment of the vehicle.

Example Scenarios

FIG. 3 is an example of detailed map information 300 for a section ofroadway including intersection 302. In this example, the detailed mapinformation 300 includes information identifying the shape, location,and other characteristics of lane lines 304, 306, traffic signal lights308, 310, crosswalks 312, 314, sidewalks 316, 318, stop sign 320, andyield sign 322. Vehicle 324 may be configured to operate in anautonomous driving mode.

Although the map information is depicted herein as an image-based map,the map information need not be entirely image based (for example,raster). For example, the map information may include one or moreroadgraphs or graph networks of information such as roads, lanes,intersections, and the connections between these features. Each featuremay be stored as graph data and may be associated with information suchas a geographic location and whether or not it is linked to otherrelated features. For example, a stop light or stop sign may be linkedto a road and an intersection, etc. In some examples, the associateddata may include grid-based indices of a roadgraph to allow forefficient lookup of certain roadgraph features.

While the detailed map information 300 may be used by the vehicle forplanning and driving operations, it may not be necessary or feasible toprovide all of the details to a passenger or other user. However,depending on the situation, certain information may be of particularrelevance to the user.

By way of example only, FIGS. 4A-B illustrate situations 400 and 410,respectively, where the vehicle is stopped at an intersection. Here, thereason for the stoppage may be information helpful to the user. Forinstance, in the first view 400 the stoppage is due to a stop sign 402,whereas in the second view 410 the stoppage is due to a traffic light412. Knowing that the vehicle is waiting for the traffic light to changecolor can indicate to the user that the vehicle will be moving shortly.In contrast, with a stop sign it may be harder to determine when thevehicle is able to proceed since that may be dependent on cross traffic.The vehicle may determine the type of signage from object recognitionfrom the on-board perception system, from the detailed map information,and/or information received from a remote system (e.g., Google Maps orWaze).

In the case of a traffic light, the perception system may be able todetermine the color or other status of the light. This information maybe communicated to in-vehicle passengers via a display mounted, e.g.,along the rear-facing side of one or both of the front row seats or viaa display on a console positioned, e.g., between or behind the front rowseats. FIG. 4C illustrates one example 420 with an in-vehicle display422 with map 424 presented thereon. The map 424 illustrates the vehicle426, with a callout 428 specifically identifying the reason why thevehicle is not moving. Here, the callout 428 includes an icon thereinshowing a stop sign. Alternatively or additionally, text or othergraphics may be presented in or adjacent to the callout to aid the riderin understanding the reason for the vehicle's current status (here,stopped at the intersection due to the stop sign). And FIG. 4Dillustrates another example 430 with an in-vehicle display 432 with map434 presented thereon. The map 434 illustrates the vehicle 436, with acallout 438 similar to callout 428. However, here the callout 438includes an icon therein showing a stop light. As above, alternativelyor additionally text or other graphics may be presented in or adjacentto the callout to aid the rider in understanding the reason for thevehicle's current status (here, stopped at the intersection due to thestop light).

Alternatively or additionally, the information may be transmitted to theuser's personal device, such as a mobile phone, smart watch, tablet PC,etc. FIGS. 5A and 5B illustrate two examples 500 and 510 in which a userdevice such as a mobile phone 502 displays certain information to theuser. As shown in FIG. 5A, the graphical interface (GUI) present a map504 with the vehicle 506 and callout 508. As with the example of FIG.4C, map 514 illustrates vehicle 516 with callout 518. In this example,callout 518 includes a stop light icon.

In one implementation, the information transmitted to the user'spersonal device originates from the vehicle. This information may berouted through a remote server, for instance as part of a fleetmanagement system. In one scenario, the server would decide whether tomessage the user and how to message the user. In another scenario, thevehicle and the server both transmit status information to the user'sdevice. This may be done in collaboration between the vehicle and theserver, or independently. For instance, the vehicle may provide one setof information regarding what the vehicle sees in its environment andhow it is responding to what it sees, while the server may provideanother set of information, such as traffic status farther along theroute or other contextual data. In these scenarios, the software (e.g.,app) running on the user's device may be configured to select whatinformation to show, and when to show it, based on the received data.One or both of the vehicle or the server may select differentcommunication strategies based on the pickup status of the user (i.e.,awaiting pickup or picked up and in the vehicle). Alternatively oradditionally, or the app or other program on the user's device mayselect different communication strategies. This may be based on thepickup status, the type(s) of information received, etc.

In addition to indicating the type of signage, the vehicle may alsocommunicate a message about its status. As seen in FIGS. 5A and 5B,respective regions 509 and 519 are arranged to provide contextualdetails to the user. For a passenger awaiting pickup, this may includethe time (or distance) to pickup, information to identify the vehicle,information about the ride requested by the user (e.g., pickup anddrop-off locations, stops along the route, user preferences, etc.) aswell as information about the user's account and/or support, forinstance from a remote assistance service associated with the vehicle.

These and other graphical interfaces may be provided on an app or otherprogram operating on the user's personal device. Depending on the typeof device and the location (e.g., external to the vehicle or within thevehicle), the vehicle's onboard computer system (control system) mayselect which information to present on the user's device and how topresent it. Similarly, when the user is in the vehicle, the onboardsystem may select what information to present, and how to present it, onthe vehicle's display device(s).

For instance, as shown in the examples of FIGS. 6A-B, the message to theuser (on the in-vehicle display 600 of FIG. 6A and the app 610 on theuser's device in FIG. 6B) is “Waiting for intersection to clear”. In theexample of FIG. 6A, the message may be presented in a selected region602 of the display as illustrated by the dashed box. The reason why thevehicle is waiting may be illustrated in this or another region, e.g.,region 604 (which shows a traffic light icon), also illustrated by adashed box. In this example, even though the light may be green, theremay be pedestrians, bicyclists or other objects in the intersection. Incontrast, while the message in FIG. 6B is also presented in a selectedregion 612 of the GUI by the app, in this case the reason (e.g., atraffic light or stop sign) may be omitted. Here, the vehicle's onboardsystem may select different information to present to the user to bepicked up that is more relevant. For instance, as shown the app mayinform the user in region 614 that he or she will be picked up in twominutes. By way of example, the onboard system may evaluate the timeand/or distance until pickup when selecting whether to present icons,text, audible information or other data to the user, and choose certaindata based on a ranked list of predicted importance to the recipient.Thus, in this case the system may rank (i) a message with the time untilpickup, (ii) an icon indicating a present action or condition of thevehicle (e.g., waiting at a stop light, stop sign, yield sign, etc.),(iii) pickup location information, and the like, and select a subset ofthe ranked information for transmission to the user's device forpresentation to the user.

The information which may be ranked by the onboard system may beobtained from a variety of sources including the vehicle's perceptionsystem, the vehicle's route planner, a machine-learning based detectionmodule, a remote assistance service, etc. By way of example, the controlsystem may include a module that receives (listens to) a variety ofinputs and then distills the inputs into a user-focused communication.For instance, one set of inputs may be produced by the perceptionsystem, such as detection of traffic lights, signage, etc. Another setof inputs may be produced by a navigation or mapping system in responseto prestored or detected information. This can include extractinginformation encoded in an electronic map (e.g., whether an upcoming roadsegment is a permanent slow zone such as a school zone, or somethingthat is dynamically detected by the perception system such as aconstruction zone). And another set of inputs may include navigationdecisions or other route planning actions. Here, for instance, a machinelearning model may detect when the vehicle is unable to immediately move(e.g., due to gridlock).

The various inputs from different onboard systems thus generate messagesin real time about various conditions and situations. This informationmay be passed across a user interface bridge (or bus). The systemlistens for the information and distills it for presentation to one ormore users via user devices and/or onboard UI components. According toone aspect, a user experience (UX) framework is generated for what kindof data is to be passed to the app on a user device, what is presentedby the vehicle directly, and what is transmitted to both the device appand the vehicle's UI system.

The framework may incorporate whether the information is directly orcontextually relevant to autonomous driving decisions. For example, theonboard system may detect a red light and, as a result, the vehiclemakes a driving-related decision to stop at the red light. In such caseshaving a high confidence of accuracy (e.g., that the traffic signal isred) and relevance (e.g., that a red light will result in a delay beforethe vehicle can proceed through the intersection), the default of theframework may be to always present information to the user regarding thedriving decision. So here, the user will receive an indication that thevehicle is stopping at a red light. In contrast, contextually relevantinformation may not explicitly be related to a current driving decisionbut can have an impact on the user. Gridlock is one example. Here, thereare one or more other vehicles in front of the vehicle. This may notchange any driving decisions, but the gridlock will likely affectarrival time at a destination. Thus, in this case, the system may electto present contextual information (e.g., informing the passengers thatthe vehicle is entering a slow zone, is currently gridlocked, etc.).This contextual information may be very important to the user so theuser can gauge the trustworthiness of the arrival time.

Rankings or thresholds may be employed by the framework when choosinghow to disseminate the information. For instance, in one scenario aranking order example would be, from highest to lowest, (1) featuresthat make the vehicle stop (e.g., red light, train on a railroadcrossing, etc.), (2) features that the vehicle predicts will cause along pause in driving (e.g., a stop sign or a yield sign at a busyintersection, unprotected left turn, etc.), (3) features that can causethe vehicle to move very slowly (e.g., a construction zone, traffic,etc.) such as at lower than a posted speed, and (4) features that maymake the vehicle deviate from a normal course of action (e.g., anemergency vehicle causing the vehicle to pull over or excessive trafficor unplanned obstacles causing the vehicle to take an alternativeroute). Time may often be a threshold considered by the framework. Forinstance, micro-hesitations (e.g., on the order of 1-10 seconds) may beless perceptible to a user, but a slightly longer delay (e.g., on theorder of 30-45 seconds) may be more apparent. Thus, in the latter casethe vehicle may inform the user about the reason for the delay, but inthe former case the reason for the delay may be omitted.

The timing may be factored in for relevance to the user and for theranking order. By way of example, the framework may include restrictionson messaging hierarchy and timing. For instance, messages that areclassified as “priority” messages may supersede one or more lowerpriority messages. In one scenario, the ranking system may be on a scaleof 1-4, with 1 being the highest priority.

The framework may also select whether additional descriptive informationmay be presented, such as including a section at the top (or bottom) ofthe app that gives more context about the current scenario/state ofoperation. By way of example, an “Approaching a red light” text stringmight be displayed in addition to a callout bubble on the map with a redlight icon. In contrast, for other signals such as a green light, thecallout bubble may be presented without a further textual description toavoid cluttering the interface. Alternatively or additionally, thevisual display may include changing the route color in the case of “slowzones”.

Thus, according to aspects of the technology, the vehicle reports (viathe monologue it generates) its current status to the user based on thevehicle's knowledge about what is going on in the immediate environment.There are several aspects to this functionality. In particular,interaction with the environment/external world as detected by thevehicle, impact of this interaction on the user, and communication ofthe information to the user (either directly from the vehicle's displaysor via an app on the user's device or both). Non-visual information mayalso be presented to the user in addition or alternatively to displayingit. By way of example, the monologue messages may be verbally spokenaloud to the user. Such voice-based messages may be available on demand,such as when the user asks what the car is doing. The same messagingarchitecture described above for visual display also applied to audiblemonologue information.

There are numerous scenarios where providing the vehicle's monologue tothe user can be beneficial. As discussed above, one main type ofscenario is the vehicle reporting its current driving status or why itis waiting to take an action. Selected examples can include thefollowing. The vehicle may identify its present location, such as “I amlocated next to the tall tree adjacent to the green car. The locationinformation can be presented from the perspective of what the user isexpected or predicted to see (e.g., based on location information fromthe user's device) and/or from the perspective of what the vehicledetects from its sensors such as a camera. Another example is a “movealong” situation, e.g., where the vehicle needs to drive around theblock. Here, the information presented may include a possible pickuplocation once the vehicle loops around, a timer indicated an expectedarrival time, landmark information to aid the user to find where thevehicle will be, etc.

Other driving-related information includes communicating what thevehicle is doing and perceiving, such as yielding to pedestrian,yielding to cyclist, driving slowly through a work zone or school zone,idling at a railroad crossing waiting for a train to pass, yielding foremergency vehicles, etc. In one example, vehicle identificationinformation (e.g., text, icons or other graphics) may be displayexternally on or by the vehicle, and such information can also bepresented via the monologue as well. In another example, the vehicle mayindicate carpooling status, such as whether other riders have beenpicked up, the next rider to be picked up, where riders are sitting orwhich seats are available, an estimated time when all of the riders willhave been picked up or dropped off, etc. Whether the user is inside thevehicle or about to enter the vehicle, the monologue may indicate thedoor lock/unlock status. This information may be presented, forinstance, when the vehicle is waiting at the pickup location or uponarrival at a drop-off location. For instance, FIG. 6C illustrates anexample 620 of monologue information presented on the user's device whenthe ride is ending. Here, the information indicates that the right siderear door will unlock when the vehicle comes to a stop. And in example630 of FIG. 6C, the monologue indicates to the rider that the right siderear door is unlocked.

Information about certain vehicle systems and components may bepresented via the monologue. For instance, this can include informingpassengers about the window status, such as which windows may be rolleddown or which are locked. The app may also allow the user to controloperation for opening and closing the windows. Similarly, the windshieldwiper status or cabin temperature may be presented to or controlled bythe passenger. Here, while the vehicle may not need to activate thewipers, the user may want to have a better view of what is happeningaround the vehicle, and so may turn the wipers on or control the wiperspeed. Similarly, the user may be able to control heating or airconditioning within the cabin, turn the defrosters on or off, etc. Inone scenario, information about whether the wipers are on could be anindicator of light rain. In another scenario, precipitation may bedetected by one or more of the vehicle sensors. So in this case themonologue may inform the user to stay inside until the vehicle arrives,or to have an umbrella ready before exiting the vehicle. External(ambient) temperature information may also be communicated, for instanceto suggest that the user bundle up before exiting the vehicle.

Yet another example involves two-way communication for pickup and/ordrop off For instance, the app on the user's device may permit the userto speak with the vehicle in order to indicate where he or she islocated to expedite pickup, or to identify a landmark at which to bedropped off. Thus, the user may take actions such as messaging thevehicle by saying “I'll be there in 1 minute” or “I need another minuteor two”. The user may also take another action such as sharing his orher exact location, or ask the vehicle to find a spot closer to user'scurrent location or that avoids an undesirable spot such as in front ofa puddle or ice patch.

A further example of a type of monologue information for presentation toa user is delivery status of a package. For instance, details aboutpackage pickup or delivery, whether the vehicle is in route, where thevehicle/package is currently located may all be presented to the user.This can give the user enhanced confidence that his or her package isbeing treated carefully and that it will arrive at the intendeddestination at a particular time.

In addition to these examples, the vehicle may also communicate otherinformation such as the route generated by the on-board planner module,whether the vehicle missed a turn and needs to adjust the route, etc. Inone example, the app may enable the user to tap the screen or otherwisequery (e.g., via an audible question or hand gesture) the onboard systemfor more details about the vehicle's status or related information.

As discussed above, many different messages can be presented to theuser, whether he or she is current in the vehicle or awaiting pickup.This information may include some or all of the following messages.Additional contextual information may be included as shown in thebrackets:

-   -   Current street name    -   Waiting for intersection to clear    -   Finding a better spot to pull over    -   Pulling over    -   Pulling over for [a moment] [______ minutes]    -   Entering [school/construction/slow/work] zone    -   Yielding to emergency vehicle [vehicle type, e.g., police car,        fire truck, ambulance]    -   Slowing down for [object] [location]    -   Avoiding [object] [traffic] [delay]    -   Stopped for railroad crossing    -   Figuring this out    -   Turning hazard lights on [due to weather condition][due to road        condition]    -   Unlocking door(s) [for a driver/for a passenger]

The information presented via the monologue can be highlighted for theuser in different ways for emphasis. By way of example, a highlighted orglowing background in the app can be used to signal that the message isfrom the vehicle. The message may flash two or more times, differentgraphics may be presented in a pattern, etc.

In a stop sign example (see, e.g., FIG. 4C or 5A), a line or pulse maytrace along the planned route towards the stop sign. This may happen asthe vehicle is within a threshold distance of the stop sign (e.g.,within 50-200 meters, 1-2 blocks, or line of sight). Alternatively oradditionally, a callout may emerge from the route on the displayed mapto highlight the stop sign. An audible cue or more detailed textualinformation may be presented alternatively or additionally with thevisual information of the callout. For example, text along an upper orlower portion of the app or audio may be presented in addition to thecallout, which says something such as “Approaching a stop sign”.

In a stop light example, the information presented may depend on thelight's status. For instance, if the light is green, merely displaying arepresentation of the green light may suffice. However, if the light isred or flashing, an icon, text or other graphical representation may beaccompanied by an audible notification. The information about the stoplight may be presented when the vehicle is within a threshold distancein the same way as the stop sign example discussed above. The stop lightgraphics may remain on the display until the vehicle reaches or passesthat location. Similar approaches may be used for other signage (e.g.,yield signs), railroad crossings, etc. As noted above, depending on thetype of light or other signage, the framework may select additionalcontextual information (e.g., for a red light) or not (e.g., for a greenlight).

For slow zones (e.g., school zones, constructions zones, etc.), themonologue may highlight the portion of the route that is within thezone. A callout or other graphic can extend from or be presentedadjacent to the center of the zone. Here, the callout may move as thevehicle moves, for instance to remain in the center of the zone untilthe displayed zone shrinks below some threshold level (e.g., the zone onthe UI is smaller than the illustrated vehicle). The user may tap on orotherwise select the graphic and receive more detailed information fromthe vehicle about the situation. And when the vehicle is determiningwhat driving decision to make (e.g., “Figuring this out”), graphicaland/or audible information may be presented to the user to indicate thestatus.

In other situations, the vehicle may require some amount of time todetermine what the next driving action will be. In this type of“Figuring this out” situation, the app or the in-vehicle display mayprovide some indication that the vehicle is “temporarily thinking”.Here, icons, text and/or audio information may indicate to the user thatthe vehicle is working on a solution. In one example, for “thinking” orpaused routes (e.g., waiting for a gridlocked intersection to clear),the route color may change to communicate temporary inactivity.

It can be seen that he vehicle's generated monologue may presentinformation graphically, for instance using easily discernible iconsand/or textual components to effectively communicate the monologueinformation to the user. The vehicle may determine how to present thisinformation to the user, and this can differ depending on whether theinformation is presented on a display within the vehicle or via an appon the user's device. By way of example, when presenting information viathe app when the user is not in the vehicle, the system may estimate orotherwise account for any delays in transmitting the data to the app.Here, the timing of status updates or specificity in the informationpresented may be changed in response to expected or detected delays insending the app the information (e.g., information about vehicle speed,stoplight status, perception information about objects in the vehicle'senvironment, etc.).

There may also be differences in how the information is sent to the appwhen the user's device is inside the vehicle and the updates can beprovided via an ad hoc communication link (e.g., Bluetooth™, BluetoothLE™, NFC) as opposed to via a cellular communication link. Finally, whenthe user is within the vehicle, the on-board system could potentiallychange the monologue appearance on the vehicle's display(s) depending ontype of device, placement of the display, where the user is sitting,and/or whether user is looking at the screen. For instance, if the useris sitting in the front passenger (or driver) seat, the system may electto present more detailed information to the user about the vehicle'scurrent status than if the user is seated behind the front row.

An example of something easier to see via the back screens (e.g.,displays positioned on the backs of the front seats or in a consolebetween the front seats) than from the front seat is a tailgater,especially if there is no display screen along the dashboard. Forinstance, in the front seat if there is no display along the dashboardarea, the passenger would need to be looking in the rear view mirror todetect another vehicle close behind. In contrast, information can bepresented on the back screen(s) showing not only the relative positionof the tailgater, but also provide information about any action theautonomous vehicle may take (or not take) in response to the othervehicle. In one scenario, with riders seated in the rear row, themonologue may not need to be displayed in the device app during a ride.However, with a rider seated in the front row, showing the monologuewould be one of the ways the vehicle may explain its behavior. There areseveral ways to detect where riders sit (e.g., an interior camera, seatoccupancy sensor, seatbelt sensor, door open indicator, etc.). Otherforms of augmenting the content may be user initiated features wheretapping the app screen (or in-vehicle screen) would trigger for themonologue to be audibly announced. There may also be specificvehicle-initiated actions, e.g., if a rider has placed items in the backrow or in the trunk, the vehicle may announce “Unlocked trunk” or“Opening rear door”.

As noted above, information may be presented to the user differentlydepending on whether he or she is viewing the information on theirdevice's app or via a cabin display. By way of example, in somesituations the app may be used for mostly or solely one-way informationprovided by the vehicle, whereas the in-cabin UI may provide interactivecapabilities for the user. Here, a monologue notification may bepresented on the screen of the user's device. The user could requestthat certain information be presented or replayed on an in-cabindisplay, such as the last 5-10 seconds of driving actions. In contrast,there may be situations where the app itself triggers a change ininformation presented by the monologue. For instance, if the person usesthe app to remotely unlock the vehicle's doors, the monologue maypresent information confirming or alerting the person to the change instatus. Another example may be as an “interplay” between the in-vehicledisplay and the app. For instance, certain monologues may be focused onexplaining the vehicle's current behavior, but some may be less clearthan others (e.g., “Figuring this out”). When these types of monologuesmight appear on the in-vehicle rider screen (or the app), the user maybe able to tap to explore or “tag” the content to be viewed later. Inthis case, the collected tags could then be relayed in follow upcommunications in the form of educational content to learn more aboutwhat the vehicle was doing or how the vehicle determines what to do.

As discussed further below, monologue information may be communicated toa user device by routing data via a cellular or other wireless link. Aremote service may help coordinate certain information communicated tothe user from the vehicle, or may provide different information to theuser and/or the vehicle. For instance, map, traffic and weather updatesmay be provided to the vehicle from the remote service. This informationmay affect the planned route, which may in turn change the statusmessage(s) sent from the vehicle to the user.

By way of example, there may be status updates or a request for actionfrom a delivery or third party service. For instance, if there iscurbside pickup of a package, the status for when something might beready for curbside delivery may be pushed (transmitted) to the in-cardisplay or the app. Schedule changes (e.g., if the trip is changed) mayalso impact routing. Thus, if a scheduled meeting on the user's calendaris canceled, this may affect a scheduled trip. This could includeevaluating a single service or a combination of services, e.g., theuser's calendar and a flight status.

The monologue may be part of a multi-layer UI stack for different typesof notifications. Other notification layers may provide information withvarious levels of importance/urgency. In one scenario, monologueinformation may be presented via a single line of text, a bubble orother callout adjacent to the vehicle (such as shown in FIGS. 4C-D), orother graphics. The information may be displayed (and/or repeatedaudibly) for as long as the event or condition is true. Thus, themessage “Yielding to cyclist” may be displayed along a portion of the UIuntil the cyclist has moved away from the vehicle or otherwise clearsfrom the vehicle's planned path. As such, information from the on-boardplanner and perception systems may be continuously evaluated todetermine the type of notification to provide and when to stop providingit.

The information, and how it is presented, may depend on what stage thetransportation status is at. For instance, the vehicle may be on the wayto a pickup, in which case the monologue information must be presentedto the user via his or her device. If the vehicle is in the process oftransporting the user to his or her destination, the information may bepresented on their device and/or the in-vehicle UI system.

FIGS. 7A-B illustrate general examples of how information may becommunicated between the vehicle and the user.

In particular, FIGS. 7A and 7B are pictorial and functional diagrams,respectively, of an example system 700 that includes a plurality ofcomputing devices 702, 704, 706, 708 and a storage system 710 connectedvia a network 712. System 700 also includes vehicles 914, which may beconfigured the same as or similarly to vehicles 100 and 120 of FIGS.1A-B and 1C-D, respectively. Vehicles 714 may be part of a fleet ofvehicles. Although only a few vehicles and computing devices aredepicted for simplicity, a typical system may include significantlymore.

As shown in FIG. 7B, each of computing devices 702, 704, 706 and 708 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto the ones described above with regard to FIG. 2 . The variouscomputing devices and vehicles may communication via one or morenetworks, such as network 712. The network 712, and intervening nodes,may include various configurations and protocols including short rangecommunication protocols such as Bluetooth™, Bluetooth LE™, the Internet,World Wide Web, intranets, virtual private networks, wide area networks,local networks, private networks using communication protocolsproprietary to one or more companies, Ethernet, WiFi and HTTP, andvarious combinations of the foregoing. Such communication may befacilitated by any device capable of transmitting data to and from othercomputing devices, such as modems and wireless interfaces.

In one example, computing device 702 may include one or more servercomputing devices having a plurality of computing devices, e.g., a loadbalanced server farm, that exchange information with different nodes ofa network for the purpose of receiving, processing and transmitting thedata to and from other computing devices. For instance, computing device702 may include one or more server computing devices that are capable ofcommunicating with the computing devices of vehicles 714, as well ascomputing devices 704, 706 and 708 via the network 712. For example,vehicles 714 may be a part of a fleet of vehicles that can be dispatchedby a server computing device to various locations. In this regard, thecomputing device 702 may function as a dispatching server computingsystem which can be used to dispatch vehicles to different locations inorder to pick up and drop off passengers or to pick up and delivercargo. In addition, server computing device 702 may use network 712 totransmit and present information to a user of one of the other computingdevices or a passenger of a vehicle. In this regard, computing devices704, 706 and 708 may be considered client computing devices.

As shown in FIG. 7A each client computing device 704, 706 and 708 may bea personal computing device intended for use by a respective user 716,and have all of the components normally used in connection with apersonal computing device including a one or more processors (e.g., acentral processing unit (CPU)), memory (e.g., RAM and internal harddrives) storing data and instructions, a display (e.g., a monitor havinga screen, a touch-screen, a projector, a television, or other devicesuch as a smart watch display that is operable to display information),and user input devices (e.g., a mouse, keyboard, touchscreen ormicrophone). The client computing devices may also include a camera forrecording video streams, speakers, a network interface device, and allof the components used for connecting these elements to one another. Asindicated in FIG. 7B, device 708 may be the device of a passenger who iscurrently in the vehicle, while device 706 may be the device of a userawaiting pickup.

Although the client computing devices may each comprise a full-sizedpersonal computing device, they may alternatively comprise mobilecomputing devices capable of wirelessly exchanging data with a serverover a network such as the Internet. By way of example only, clientcomputing devices 706 and 708 may be mobile phones or devices such as awireless-enabled PDA, a tablet PC, a wearable computing device (e.g., asmartwatch), or a netbook that is capable of obtaining information viathe Internet or other networks.

In some examples, client computing device 704 may be a remote assistanceworkstation used by an administrator or operator to communicate withpassengers of dispatched vehicles, or users awaiting pickup. Althoughonly a single remote assistance workstation 704 is shown in FIGS. 7A-7B,any number of such workstations may be included in a given system.Moreover, although workstation 704 is depicted as a desktop-typecomputer, the workstation 704 may include various types of personalcomputing devices such as laptops, netbooks, tablet computers, etc.

Storage system 710 can be of any type of computerized storage capable ofstoring information accessible by the server computing devices 702, suchas a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, flash drive and/ortape drive. In addition, storage system 710 may include a distributedstorage system where data is stored on a plurality of different storagedevices which may be physically located at the same or differentgeographic locations. Storage system 710 may be connected to thecomputing devices via the network 712 as shown in FIGS. 7A-B, and/or maybe directly connected to or incorporated into any of the computingdevices.

In a situation where there are passengers, the vehicle or remoteassistance may communicate directly or indirectly with the passengers'client computing device. Here, for example, information may be providedto the passengers regarding current driving operations, changes to theroute in response to the situation, etc. As explained above, informationmay be passed from the vehicle to the passenger or other user via thevehicle's monologue UI. For instance, when the user is awaiting pickup,the vehicle may send monologue information via network 712. However,when the vehicle arrives at the pickup location or the user enters thevehicle, the vehicle may communicate directly with the user's device,e.g., via a Bluetooth™ or NFC communication link. Communication delays(e.g., due to network congestion, bandwidth limitations, coverage deadzones, etc.) may be factored in by the vehicle when deciding whatspecific information is provided by the monologue.

FIG. 8 illustrates a method of operation in accordance with theforegoing. At block 802, the method includes determining, by one or moreprocessors of a vehicle operating in an autonomous driving mode, statusinformation associated with the vehicle. At block 804, a pickup statusof a customer is determined by the processors. The pickup statusindicates whether the customer is awaiting pickup by the vehicle or iscurrently being driven by the vehicle in the autonomous driving mode. Atblock 806, the one or more processors select one of a plurality ofcommunication options for communication with the customer based on thepickup status. At block 808, the processor(s) generates, in accordancewith the selected communication option, a monologue message indicatingstatus information associated with the vehicle. And at block 810 themonologue message is provided for presentation to the customer.

FIG. 9 illustrates another method of operation in accordance with theforegoing. At block 902 the method includes determining, by one or moreprocessors, status information associated with a vehicle operating in anautonomous driving mode. At block 904, the one or more processors,determine a pickup status of a customer. The pickup status indicateswhether the customer is awaiting pickup by the vehicle or is currentlybeing driven by the vehicle in the autonomous driving mode. At block906, the one or more processors select, based on the pickup status, oneof a plurality of communication options for communication with thecustomer via a personal communication device of the customer. At block908, the one or more processors generate, in accordance with theselected communication option, a monologue message indicating statusinformation associated with the vehicle. And a block 910, the monologuemessage is provided for presentation to the customer via the personalcommunication device.

In one example, a first one of the one or more processors may be aprocessing device of the vehicle or processing device of a remote serverin operative communication with the vehicle. In another example, a firstone of the one or more processors may be a processing device of thevehicle while a second one of the one or more processors is a processingdevice of the vehicle. In this case, the method includes the firstprocessor generating a first set of information for the monologuemessage the second processor generating a second set of information forthe monologue message. And in a further alternative, a first one of theone or more processors is a processing device of the personalcommunication device. Here, the method includes the processing deviceselecting information received from at least one of the vehicle or aremote server for presentation in the monologue message.

Finally, as noted above, the technology is applicable for various typesof vehicles, including passenger cars, buses, RVs and trucks or othercargo carrying vehicles.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements. Theprocesses or other operations may be performed in a different order orsimultaneously, unless expressly indicated otherwise herein.

1. A method comprising: causing, by one or more processors, a vehicle tooperate along a route towards a location in an autonomous driving mode;obtaining, by the one or more processors, sensor data from a perceptionsystem of the vehicle, the sensor data indicating one or more objects inan external environment of the vehicle along the route; determining, bythe one or more processors based on the obtained sensor data, a currentdriving situation; receiving, by the one or more processors, contextualdata regarding the route; generating, by the one or more processors, adriving decision based on one or more of the obtained sensor data, thecurrent driving situation, and the contextual data; operating thevehicle in the autonomous driving mode according to the drivingdecision; generating, by the one or more processors, a monologue messageindicating current status information corresponding to the drivingdecision; and causing, by the one or more processors, the monologuemessage to be presented to a user of the vehicle to indicate to the usera reason for the driving decision.
 2. The method of claim 1, whereincausing the monologue message to be presented to the user includesselecting to present the monologue message via at least one of a userinterface system of the vehicle or a client device of the user.
 3. Themethod of claim 1, wherein the current driving situation is a stoppedsituation and the current status information explains why the vehiclehas stopped.
 4. The method of claim 3, wherein the monologue messagevisually indicates the stopped situation involves a stop sign, a stoplight, or a railroad crossing.
 5. The method of claim 3, wherein themonologue message visually indicates the stopped situation involvesgridlock or an unprotected turn.
 6. The method of claim 1, wherein thecurrent driving situation is a slowed situation and the current statusinformation explains the slowed situation.
 7. The method of claim 6,wherein the monologue message visually indicates the slowed situationinvolves yielding to a road user.
 8. The method of claim 6, wherein themonologue message visually indicates the slowed situation involves aslow zone.
 9. The method of claim 1, wherein causing the monologuemessage to be presented to the user includes selecting to present themonologue message via one or more display devices of a user interfacesystem of the vehicle.
 10. The method of claim 9, wherein: generatingthe monologue message indicating current status informationcorresponding to the driving decision includes selecting a set of dataabout the driving decision; and causing the monologue message to bepresented includes selecting a first subset of the set of data forpresentation on a first one of the one or more display devices, andselecting a second subset of the set of data for presentation on asecond one of the one or more display devices.
 11. The method of claim10, wherein: the first display device is located along a first areawithin a cabin of the vehicle; the second display device is locatedalong a second area within the cabin; the first subset includes a firstamount of detail about the current status information; and the secondsubset includes a second amount of detail about the current statusinformation, the second amount of detail being less than the firstamount of detail.
 12. The method of claim 1, wherein the contextual datais received from a remote system.
 13. The method of claim 12, whereinthe contextual data indicates a traffic status along an upcoming portionof the route.
 14. A vehicle operating in an autonomous driving mode, thevehicle comprising: a perception system; and one or more processorsoperably coupled with the perception system, the one or more processorsconfigured to: cause the vehicle to operate along a route towards alocation in the autonomous driving mode; obtain sensor data from theperception system, the sensor data indicating one or more objects in anexternal environment of the vehicle along the route; determine, based onthe obtained sensor data, a current driving situation; receivecontextual data regarding the route; generate a driving decision basedon one or more of the obtained sensor data, the current drivingsituation, and the contextual data; operate the vehicle in theautonomous driving mode according to the driving decision; generate amonologue message indicating current status information corresponding tothe driving decision; and cause the monologue message to be presented toa user of the vehicle to indicate to the user a reason for the drivingdecision.
 15. The vehicle of claim 14, wherein the one or moreprocessors are further configured to select to present the monologuemessage via at least one of a user interface system of the vehicle or aclient device of the user.
 16. The vehicle of claim 14, wherein thecurrent driving situation is a stopped situation and the current statusinformation explains why the vehicle has stopped.
 17. The vehicle ofclaim 16, wherein the monologue message visually indicates the stoppedsituation involves a stop sign, a stop light, or a railroad crossing.18. The vehicle of claim 16, wherein the monologue message visuallyindicates the stopped situation involves gridlock or an unprotectedturn.
 19. The vehicle of claim 14, wherein the current driving situationis a slowed situation and the current status information explains theslowed situation.
 20. The vehicle of claim 19, wherein the monologuemessage visually indicates the slowed situation involves yielding to aroad user or involves a slow zone.